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Abstract 

We discuss recent developments on branching problems of irre- 
ducible unitary representations ir of real reductive groups when re- 
stricted to reductive subgroups. Highlighting the case where the un- 
derlying (a, i'Q-modules of tt are isomorphic to Zuckerman derived 
functor modules AJX), we show various and rich features of branching 
laws such as infinite multiplicities, irreducible restrictions, multiplicity- 
free restrictions, and discrete decomposable restrictions. We also for- 
mulate a number of conjectures. 
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1 Introduction 



Zuckerman derived functor is powerful algebraic machinery to construct ir- 
reducible unitary representations by cohomological parabolic induction. The 
(5, iT)-modules A q (X), referred to as Zuckerman derived functor modules, 
give a far reaching generalization of the Borei-Weil-Bott construction of ir- 
reducible finite dimensional representations of compact Lie groups. They 
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include Harish-Chandra's discrete series representations of real reductive Lie 
groups as a special case, and may be thought of as a geometric quantization 
of elliptic orbits (see Fact 16. ip . 

Branching problems in representation theory ask how irreducible repre- 
sentations 7T of a group G decompose when restricted to a subgroup G '. 

The subject of our study is branching problems with emphasis on the set- 
ting when (G,G') is a reductive symmetric pair (Subsection I2.3p . and when 
7r is the unitarization of a Zuckerman derived functor module A q (\). We see 
that branching problems in this setting include a wide range of examples: a 
very special case is equivalent to finding the Plancherel formula for homo- 
geneous spaces (e.g. Proposition 12.41 and Example 14. S|) and another special 
case is of combinatorial nature (e.g. the Blattner formula). 

In this article, we give new perspectives on branching problems by reveal- 
ing the following surprisingly rich and various features: 

• The multiplicities may be infinite (Section [2} and may be one (Section 

• The restriction may stay irreducible (Section [3]). 

• The spectrum may be purely continuous and may be discretely decom- 
posable (Section [3} . 

Finally, we present a number of open problems that might be interesting 
for further study (see Conjectures 14.21 14.31 15.41 and 15.111) . 

This article is based on the talk presented at the conference "Representa- 
tion Theory and Mathematical Physics" in honor of Gregg Zuckerman's 60th 
birthday at Yale University on October 2009. The author is one of those who 
have been inspired by Zuckerman's work, and would like to express his sin- 
cere gratitude to the organizers of the stimulating conference, Professors J. 
Adams, M. Kapranov, B. Lian, and S. Sahi for their hospitality. 

2 Wild aspects of branching laws 
2.1 Analysis and synthesis 

One of the most distinguished feature of unitary representations is that they 
are always built up from the smallest objects, namely, irreducible ones. For 
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a locally compact group G, we denote by G the set of equivalence classes of 
irreducible unitary representations of G, endowed with the Fell topology. 

Fact 2.1 (Mautner-Teleman). Every unitary representation n of a locally 
compact group G is unitarily equivalent to a direct integral of irreducible 
unitary representations: 



Here, dfi is a Borel measure on G, n n : G — > N U {00} is a measurable 
function, and n n (cr)a stands for the multiple of an irreducible unitary repre- 
sentation a with multiplicity n n (a). 

The decomposition (12. ip is unique if G is of type I in the sense of von 
Neumann algebras. Reductive Lie groups are of type I. Then the multiplicity 
function n n is well-defined up to a measure zero set with respect to d[i. We 
say that n has a uniformly bounded multiplicity if there is C > such that 
n^(a) < C almost everywhere; ir is multiplicity-free if n w (o~) < 1 almost 
everywhere, or equivalently, if the ring of continuous G-endomorphisms of 7r 
is commutative. 

2.2 Branching laws and Plancherel formulas 

Suppose that G' is a closed subgroup of G. Here are two basic settings where 
the problem of decomposing unitary representations arises naturally. 

1) (Induction G' t G) Plancherel formula. 

For simplicity, assume that there exists a G-invariant Borel measure on 
the homogeneous space G/G'. Then the group G acts unitarily on the 
Hilbert space L 2 (G/G') by translations. The irreducible decomposition 
of the regular representation of G on L 2 (G/G') is called the Plancherel 
formula for G/G' . 

2) (Restriction G 1 G') Branching laws. 

Given an irreducible unitary representation tt of G. By the symbol 
7t\g>, we think of tt as a representation of the subgroup G' . The branch- 
ing law of the restriction it\g> means the formula of decomposing 7r 
into irreducible representations of G' . Special cases of branching laws 




(2.1) 
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include the classical Clebsch-Gordan formula, or more generally, the 
decomposition of the tensor product of two irreducible representations 
(fusion rule), and the Blattner formula, etc. 

2.3 Symmetric pairs 

We are particularly interested in the branching laws with respect to reductive 
symmetric pairs. Let us fix some notation. 

Suppose a is an involutive automorphism of a Lie group G. We denote 
by G u := {g G G : og = g}, the group of fixed points by a. We say that 
(G, G') is a symmetric pair if G' is an open subgroup of G a . Then the 
homogeneous space G/G' becomes an affine symmetric space with respect to 
the canonical G-invariant affine connection. The pair (G, G') is said to be 
a reductive symmetric pair if G is reductive. Further, if G' is compact then 
G/G' becomes a Riemannian symmetric space. 

Example 2.2. 1) (group case) Let V G be a Lie group, G :=G x'G the di- 
rect product group, and a G Aut(G) be defined as a(x,y) := (y,x). Then 
G a = diag('G) := {(x, x) : x G'G}. Since the homogeneous space G/G a is dif- 
feomorphic toG, we refer to the symmetric pair (G, G u ) = ( S G x"G, diag('G)) 
as a group case. 

2) The followings are chains of reductive symmetric pairs: 

GL(2n, H) D GL(n, C) D GL(n, R) D GL(p, K) x E) (p + g = n), 

0(4p,4g) D U(2p, 2q) D Sp(p,q) D U(p,q) D 0(p,q). 

2.4 Finite multiplicity theorem of van den Ban 

Let (G, G') be a reductive symmetric pair. 

The irreducible decomposition ( 12.1 j) is well-behaved for the induction G" t 
G, namely, for the Plancherel formula of the symmetric space G/G': 

Fact 2.3 (van den Ban |2J). Suppose (G,G') is a reductive symmetric pair. 
Then the regular representation ir on L 2 (G/G') has a uniformly bounded 
multiplicity. 

2.5 Plancherel formulas v.s. branching laws 

Fairly many cases of the Plancherel formula for L 2 (G/G') treated in Fact 12.31 
can be realized as a special example of branching laws of the restriction of 
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irreducible unitary representations of other groups. For example, we recall 
from [2T] Propositions 6.1, 6.2] and [2"5| Theorem 36]: 

Proposition 2.4. Let G/G' be a reductive symmetric space. Then the regular 
representation of G on L 2 (G/G') is unitarily equivalent to the restriction 
7t|c; for some irreducible unitary representation it of a reductive group G 
containing G as its subgroup if (G, G') fulfills one of the following conditions: 

(A) G' is compact and the crown domain D of the Riemannian symmetric 
space G/G' is a Hermitian symmetric space, 

or 

(B) G'/Zq has a split center. Here Zq stands for the center of G. 

Remark 2.5. 1) Most Riemannian symmetric pairs (G,G') satisfy the as- 
sumption (A) (see [37] for details). 

2) As the proof below shows, 

GdGdG' 
is a chain of reductive symmetric pairs. 

3) We can take ir to be the unitarization of some AJX) in (A) and also 
in (B) when G is a complex reductive Lie group. 

4) There are some more cases other than (A) or (B) for which the con- 
clusion of Proposition 12.41 holds. For instance, see Example 14.81 for the 

group case L 2 (GL(n, C)) and also for a more general case L 2 (GL(2n, M.)/GL(n, C)). 

Outline of the proof. The choice of tt and G depends on each case (A) and 
(B). 

(A) We take G to be the automorphism group of D, and 7r to be any holo- 
morphic discrete series representation of G of scalar type. Then 7r is realized 
in the Hilbert space consisting of square integrable, holomorphic sections of 
a G-equivariant holomorphic line bundle over D. Since holomorphic sections 
are determined uniquely by the restriction to the totally real submanifold 
G/G', we get a realization of the restriction tt\g in a certain Hilbert subspace 
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of A(G/G'), which itself is not L 2 (G/G') but is unitarily equivalent to the 
regular representation on L 2 {G/G') (see [T2]). 

(B) Let P be a maximal parabolic subgroup of G whose Levi part is 
G' . Take G to be the direct product G x G, and n to be the outer tensor 
product representation m M 712 where tti is a degenerate unitary principal 
series representation induced from a unitary character of P and 7r 2 is the 
contragredient representation of Hi. Then apply the Mackey theory. □ 

Example 2.6. 1) The regular representation on L 2 (G/G') = L 2 (GL(n,R)/0{n)) 
is unitarily equivalent to the restriction of a holomorphic discrete series rep- 
resentation of G := Sp(n,R) to G. 

2) The regular representation on L 2 (GL(n, R) /GL(p, R) x GL(q, R)) with 
(p + q = n) is unitarily equivalent to the restriction of a degenerate principal 
representation of G := GL(n,R) x GL(n,R) to G (namely, to the tensor 
product representation). 

2.6 Wild aspects of branching laws 

Retain our assumption that (G, G') is a reductive symmetric pair. 

Proposition 12.41 suggests that branching problems include a wide range of 
examples. In fact, while the 'good behavior' in Fact 12.31 for the Plancherel 
formula of the symmetric space G/G', the branching law of the restriction 
7t|g" does not behave well in general. Even when ttk is a Zuckerman derived 
functor module A q (X), we cannot expect: 

'False Theorem' 2.7. Let (G, G') be a reductive symmetric pair, and 7r an 
irreducible unitary representation ofG. Then the multiplicities of the discrete 
spectrum in the branching laws tt\g' are finite. 

Remark 2.8. Such a multiplicity theorem holds for reductive symmetric pairs 
(G,G') under the assumption that the restriction tt\g> is infinitesimally dis- 
cretely decomposable in the sense of Definition 15.31 (cf. [22| [28]). A key to 
the proof is Theorem 15.61 on a criterion of i^'-admissibility and Corollary 
15.81 on an estimate of the associated variety. See Remark 15.141 for the case 

TT K ~ A q (A). 

Before giving a counterexample to (false) 'Theorem' 12.71 about the dis- 
crete spectrum, we discuss an easier case, namely, an example of infinite 
multiplicities in the continuous spectrum of the branching law: 
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Proposition 2.9 (G X G 4- diagG ). (Gelfand-Graev [8].) If tci and n 2 

are two unitary principal series representations of G = SL(n,C) (n > 3), 
then the multiplicities in the decomposition of the tensor product tti £g> 1x2 are 
infinite almost everywhere with respect to the measure dfj, in the direct integral 

(EU). 

We recall the underlying ($3, f^-modules of unitary principal series rep- 
resentations of a complex reductive Lie group are obtained as a special case 
of Zuckerman derived functor modules A q (X). 

Hence we get 

Observation 2.10. The multiplicities of the continuous spectrum in the 
branching law of the restriction tt\g' may be infinite even in the setting where 
tck — A q (X) and (G,G') is a reductive symmetric pair. 

Here is a more delicate example, which yields a counterexample to (false) 
'Theorem 12.71 about the discrete spectrum. 

Proposition 2.11 (G<c I Gr). (see [26]) There exist an irreducible unitary 
principal series representation n of G = SO(5,C) and two irreducible uni- 
tary representations T\ (a holomorphic discrete series representation) and 
T2 (a non-holomorphic discrete series representation) of the subgroup G' = 
SO(3, 2) such that 

< dimHoniG'(Ti, ir\a>) < 00 and dim Rome ( t 2 , tt\g>) = 00. 

Here, Hohig</(-, •) denotes the space of continuous G' -intertwining operators. 

3 Almost irreducible branching laws 

Let G be a real reductive Lie group, G' a subgroup, and n an irreducible 
unitary representation of G. 

We have seen some wild aspects of branching laws in the previous section. 
As its opposite extremal case, this section highlights especially nice cases, 
namely, where the restriction h\g' remains irreducible or almost irreducible 
in the following (obvious) sense: 

Definition 3.1. We say a unitary representation 7r is almost irreducible if tt 
is a finite direct sum of irreducible representations. 
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It may well happen that the restriction tc\g> is almost irreducible when G' 
is a maximal parabolic subgroup of G, but is a rare phenomenon when G' is 
a reductive subgroup. Nevertheless, we find in Subsections 13 .214331 that there 
exist a small number of examples where the restriction tt\g> stays irreducible, 
or is almost irreducible in some cases. 

We divide such irreducible unitary representations it of G into three cases, 
according as are Zuckerman derived functor modules A q (X) (see Theorem 
13. 5p . principal series representations (see Theorem I3.8[) . and minimal repre- 
sentations (see Theorem 13 .lip . From the view point of the Kostant-Kirillov- 
Duflo orbit method, they may be thought of as the geometric quantization 
of elliptic, hyperbolic, and nilpotent orbits, respectively. 

3.1 Restriction to compact subgroups 

First of all, we observe that almost irreducible restrictions tt\g> happen only 
when G' is non-compact if dini7r = oo. 

Let K be a maximal compact subgroup of a real reductive Lie group G. 

Observation 3.2. For any irreducible infinite dimensional unitary repre- 
sentation tt of G, the branching law of the restriction tt\x contains infinitely 
many irreducible representations of K . 

Proof. Clear from Harish-Chandra's admissibility theorem (see Fact 13.41 be- 
low). □ 

For later purpose, we introduce the following terminology: 

Definition 3.3. Suppose K' is a compact group and n is a representation 
of K' . We say tt is fC'-admissible if dim Horner, tv) < oo for any r e K'. 

With this terminology, we state: 

Fact 3.4 (Harish-Chandra's admissibility theorem). Any irreducible unitary 
representation tt of G is K -admissible. 

We shall apply the notion of fT'-admissibility when K' is a subgroup of K, 
and see that it plays a crucial role in the theory of discretely decomposable 
restrictions in Section |5j 
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3.2 Irreducible restriction tt\g' with ttk = A q (\) 

This subsection discusses for which triple (G,G',ir) the restriction tt\g> is 
(almost) irreducible in the setting that the underlying (g, i^)-module hk is 
isomorphic to a Zuckerman derived functor module A, (A). 

Let q be a ^-stable parabolic subalgebra of 0c = ®u C, L := A^ G (q) = 
{g £ G : Ad(g)q = q}, and A q (X) the unitary representation of G whose 
underlying (g, i^)-module is A q (X). 

Theorem 3.5 (|19j). Suppose that (G,G',L) is one of the following triples: 



G 


G' 


L 


SU (n, n) 


Sp(n, R) 


U(n — 1, n) 


SU(2p,2q) 


Sp(p, q) 


U(2p-l,2q) 


SO (2p,2q) 


SO (2p,2q-l) 


U (p, q) 


5O (4,3) 


G 2 (R) 


5O (4, 1) x SO (2) 


5O (4,3) 


G 2 (R) 


50(2) x 5O (2,3) 


SL(2n, C) 


Sp(n, C) 


GL(2n-l,C) 


SO(2n,C) 


50(2n-l,C) 


GL(n,C) 


50(7, C) 


G 2 (C) 


C x x 50(5, C) 


SU(2n) 


Sp(n) 


U(2n-1) 


S0{2n) 


S0{2n - 1) 


U(n) 


SO{7) 


G 2 ^ C ompact 


50(2) x 50(5) 



Then, the restriction Aq(X)\a' is almost irreducible for any X satisfying the 
positivity and integrality condition (see Subsection \6'.m\) . Further, the restric- 
tion A q (X)\c' stays irreducible if the character A|p of I' :— g'fl I is in the good 
range with respect to q' := g c flq (see (16. 2\f ). On the level of Harish- Chandra 
modules, we have an isomorphism 

A q (A)~A,,(A|,), 

as (g' , K') -modules. 

Outline of proof. We recall the following well-known representations of spheres: 

Sp{n)/Sp{n - 1) ^ U(2n)/U(2n - 1) ~ 5 4 "" 1 , 
U{n)/U{n - 1) A SO(2n)/SO(2n - 1) ~ 5 2n "\ 
Spin(5)/Spin(3) ^ Spin(7)/G 2 ~ 5 7 . 
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Then the trick in [20, Lemma 5.1] shows that the natural inclusion map 
G'/L' ■=->■ G/L is in fact surjective for any of the specific triples (G, G', L) in 
Theorem 13.51 where we set V := G'C\L. Further, we have flc + l = flc so that 
the inclusion q' c <^-> $jc induces the bijection q' c /c\' ^> 0c/q and L' coincides 
with Ng> {({')■ Thus, the diffeomorphism G'/L' ^> G/L is biholomorphic. In 
turn, we get an isomorphism of canonical line bundles (see (16. ip ): 

G' x L > C 2p ( u ') -> G x L C 2 p( u ) 
I I 
G'/L' ^ G/L 

This implies 

p(u)\ v =p(u') 

in the setting of Theorem 13.51 Let C\ be a 0-equivariant holomorphic line 
bundle over G/L for A G Then the pull-back of £\ + 2 P ( u ) to G'/L' 

yields a G'-equivariant holomorphic line bundle £al+2p(u') over G'/L'. Hence, 
we have natural isomorphisms 

H§{G/L,C x+ 2 P (u)) ^ H§(G'/L', A| ( ,+2 P (u')) 

between Dolbeault cohomology groups. Thus, we get Theorem 13.51 in view 
of the geometric interpretation of Zuckerman derived functor modules (see 
Section [H]). □ 

Remark 3.6. 1) The pairs (G,G') in Theorem 13.51 are reductive symmetric 
pairs except for the case (G,G f ) = (£0o(4, 3), G2OR)). 

2) The pair (g, () is a reductive symmetric pair in all the cases of Theorem 
13.51 (q is of symmetric type in the sense of Definition 14. ip . Correspondingly 
there are two choices of ^-stable parabolic subalgebras q of Qc with N G (q) ~ 
L. In either case, Aq(X)\a> is almost irreducible. 

3) In the compact case (i.e. the last three rows), the restriction is irre- 
ducible for all A. 

Example 3.7. 1) In [20] we gave a different proof of Theorem 13.51 for the 
pair 500(4, 3) 4- 02 (M) based on the Beilinson-Bernstein localization theory, 
and then applied it to construct (all) discrete series representations for non- 
symmetric homogeneous spaces 02 0R)/iSX(3,R) and 02(M)/S'£/(2, 1). 

2) H. Sekiguchi applied the restriction of A q (A) with respect to the sym- 
metric pair U(n, n) l Sp(n, R) for more general q to get a range characteri- 
zation theorem of the Penrose transform (see [H]). Following the notation in 
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[H], Proposition 1.5], we see that the unitary character C\ is in the weakly fair 
range for the ^-stable maximal parabolic subalgebra q considered in Theorem 
13.51 if and only if A = Aiei with Ai > —n. Further, A q (X) is irreducible as a 
u(n, n)-module for all Ai > — n. Its restriction to sp(n, E) stays irreducible for 
Ai > — n, but splits into two irreducible modules (W(n, l)+)K®{W{n, l)-)jf. 

3) Dunne and Zierau [6] determined the automorphism groups of elliptic 
orbits. It follows from their results that our list in Theorem 13.51 exhausts all 
the cases where Aq(A)|c stays irreducible for sufficiently positive A. 

3.3 Irreducible restriction 7t\c with it = Indp(r) 

This subsection discusses for which triples {G,G',ix) the restriction 7t\g' is 
(almost) irreducible in the setting that ir is a (degenerate) principal series 
representation ir = Indp(r) of G. 

Let P be a parabolic subgroup of G with Levi decomposition P = LN. 
For an irreducible unitary representation r of L, we extend it to P by letting 
N act trivially, and denote by Indp(r) the unitarily induced representation 
ofC. 

Theorem 3.8. Suppose that (G, G', L) is one of the following triples: 



G 


G' 


L 


SL(2n, C) 


Sp(n, C) 


GL{2n-l,C) 


SO(2n,C) 


SO{2n -1,C) 


GL(n,C) 


50(7, C) 


2 (C) 


C x x 50(5, C) 


SL(2n, R) 


Spin, E) 


OL(2n-l,E) 


SO(n, n) 


SO(n,n-l) 


GL(n, E) 


50(4,3) 


G 2 (E) 


50(1,1) x 50(3,2) 



Then, the degenerate unitary principal series representations n = Ind P (r) 
of G are almost irreducible when restricted to the subgroup G' for any one 
dimensional unitary representation r of any parabolic subgroup P having L 
as its Levi part. 

Outline of the proof. The subgroup G' acts transitively on the (real) flag va- 
riety G/P in the setting of Theorem 13.81 and the isotropy subgroup P' : = 
G'flP becomes a parabolic subgroup of G'. Then we get an isomorphism 
G'/P' G/P, and hence the conclusion follows. □ 

We note that the parabolic subgroup P in Theorem 13.81 is maximal. 
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Example 3.9. For simplicity, we use GL(2n, M) instead of the semisimple 
group SL(2n, R) in the fourth row, and consider the reductive symmetric pair 
(G,G r ) = (GL(2n, R), Sp(n, R)). Let P be a maximal parabolic subgroup P 
of G with Levi subgroup L = GL(2n - 1,R) x GL(1,R). Then P has an 
abelian unipotent radical R 2n_1 and P' = G'dP has a non-abelian unipotent 
radical which is isomorphic to the Heisenberg group H 2n ~ x . In this case the 
unitary representation ir = Indp(r) is irreducible as a representation of G for 
any unitary character r of P. On the other hand, the restriction of 7r to G' is 
more delicate. It stays irreducible for generic r (i.e. dr ^ 0) and splits into 
two irreducible representations of G' for singular r, giving rise to a 'special 
unipotent representation' of G' = Sp(n,~R). See |35J for a detailed analysis 
in connection with the Weyl operator calculus. 

Remark 3.10. For a complex reductive group, the underlying (g, i^)-modules 
of (degenerate) principal series representations are isomorphic to some A q (\). 
Thus the first three cases in Theorem 13.81 have already appeared in Theorem 
13.51 in the context of A q (\). 

3.4 Irreducible restriction of minimal representation 

Thirdly, we present an example of almost irreducible branching laws for rep- 
resentations 7r which are supposed to be attached to minimal nilpotent coad- 
joint orbits. 

Let w be the irreducible unitary representation of the indefinite orthogo- 
nal group G — 0(p, q) for p, q > 2, (p, q) ^ (2, 2) and p + q even, constructed 
in [3] or [3l| Part I]. It is a representation of Gelfand-Kirillov dimension 
p + q — 3, and is minimal in the sense that its annihilator in the enveloping 
algebra U (g) is the Joseph ideal if p + q > 6. 

Theorem 3.11 (0(p, q) I 0{p, q-l)). 

™\o<jm-\) — y+ + VI 

where V± are irreducible representations of 0(p,q — 1). 

Proof. See [331 Corollary 7.2.1]. □ 

Remark 3.12. The irreducible decomposition V+ + V- has a geometric mean- 
ing in connection to the smallest L 2 -eigenvalues of the (ultra-hyperbolic) 
Laplacian on pseudo-Riemannian space forms. 
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4 Multiplicity- free conjecture 



Irreducible restrictions to reductive subgroups are a somewhat rare phe- 
nomenon, as we have seen in the previous section. On the other hand, it 
happens more often that the restriction is multiplicity-free with respect to 
reductive symmetric pairs (G,G') (see [29] for examples). In this section, 
we propose a conjectural sufficient condition for the restriction tt\g' to be 
multiplicity-free in the setting where hk is a Zuckerman derived functor 
module ^4 q (A). Our conjecture is motivated by the propagation theorem of 
multiplicity-free property under 'visible actions' [31]. 

Definition 4.1. 1) We say a ^-stable parabolic subalgebra q = [ + u is of 
symmetric type if (g, I) forms a symmetric pair. 

2) We say that q is of virtually symmetric type if there exists a ^-stable 
parabolic subalgebra q of symmetric type such that L/L = N G (q)/N G (q) is 
compact. 

Remark. 1) If q is of virtually symmetric type, then we have a fibration 
L/L^G/L^G/L with compact fiber L/L. 

2) If q is of symmetric type, then q is obviously of virtually symmetric 
type. 

3) Any parabolic subalgebra is of virtually symmetric type if G is com- 
pact. 

Let q be a ^-stable parabolic subalgebra of gc, and A q (\) be the unita- 
rization of A q (X). Suppose (g, g') is a reductive symmetric pair. We then 
propose the following two conjectures: 

Conjecture 4.2. If a 6-stable parabolic subalgebra q is of symmetric type, 
then the restriction AJ\)\g' is multiplicity-free for sufficiently regular A. 

Conjecture 4.3. If q is of virtually symmetric type, then the restriction 
A|(A)|g' has a uniformly bounded multiplicity. 

Here are some affirmative cases: 

Example 4.4. Suppose G is a non-compact simple Lie group such that 
G/K is a Hermitian symmetric space. We write g = t + p for the Cartan 
decomposition. Then pc := p ®r C decomposes into a direct sum of two 
irreducible representations of K, say pc = p + © p_. Then q := t c + p + is a 
^-stable parabolic subalgebra of symmetric type. If A is in the good range, 
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then A q (X) is the underlying (g, if)-module of a holomorphic discrete series 
representation of scalar type. In this case, we see Conjecture 14.21 holds by 
the explicit branching law: 

G' = K ■ ■ ■ Hua H3|, Kostant, Schmid [ffi], 

G' : non-compact • • • Kobayashi [29J . 

Example 4.5. As a generalization of Example I4.4[ we retain that G/K is 
a Hermitian symmetric space, and assume that q is of holomorphic type in 
the sense that q n pc D P+- Then A q (X) is at most a finite direct sum of 
irreducible unitary highest weight modules if A is in the weakly fair range 
(see PQ). In this case, Conjecture 14.31 is true for any A q (\) (see [301 Theorem 
B]). Further, it was proved in [301 Theorems A, C] as a special case of the 
propagation theorem of multiplicity- free property that the restriction tt\g' is 
multiplicity-free if 7r is an irreducible unitary highest weight module of scalar 
type. 

Example 4.6. For (G, G') = (0(p, q),0(r) x 0(p — r, q)) and for a ^-stable 
parabolic subalgebra q of maximal dimension, we see from explicit branching 
laws [18] that Conjecture 14.21 holds in this case. Likewise, Conjecture 14.21 
holds for the restriction 0(2p, 2q) I U(p, q) again by explicit branching laws 

m- 

Example 4.7. For any compact group G, the restriction 7Tx\g' is always 
multiplicity-free if q is of symmetric type ([501 Theorems E, F]) and hence, 
Conjecture 14.21 is true. 

Example 4.8. Let (G, G') = (GL(2n, C), GL(n, C) x GL(n,C)), and q a 
6*-stable parabolic subalgebra such that N G (q) ~ G 1 . Then q is of symmetric 
type. Further, we have the following unitary equivalence: 

A q (\)\ G L(n,c)xGL(n,c) - L 2 (GL(n, C)). 

Thanks to the Plancherel formula of the group GL(n, C) due to the Gelfand 
school and Harish-Chandra, we see that Conjecture 14.21 holds also in this 
case. 

Let us retain the same ^-stable parabolic subalgebra q and consider an- 
other reductive symmetric pair (G, G") = (GL(2n, C), GL(2n, R)). Then, we 
get the following unitary isomorphism: 

MXj\ GL(2nM) ~ L 2 (GL(2n,R)/GL(n,C))- 
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Again, the right-hand side is multiplicity-free by the Plancherel formula for 
reductive symmetric space due to Oshima, van den Ban, Schlichtkrull, and 
Delorme [I] among others. (It should be noted that the Plancherel formula 
for a reductive symmetric space is not multiplicity-free in general.) 

Remark 4.9. As we have seen in Example 14. 81 Conj ect ures 14 . 2 1 and 14 . 3 1 refer to 
the multiplicities in both discrete and continuous spectrum in the branching 
law A q (\)\c>- 

5 Discretely decomposable branching laws 

This section highlights another nice class of branching problems, namely, 
when the restriction -k\q> splits discretely without continuous spectrum. 

An obvious case is when dim7r < oo or when G' is compact. One of 
the advantages of discretely decomposable restrictions is that we can expect 
a combinatorial and detailed study of branching laws by purely algebraic 
methods because we do not have analytic difficulties arising from continuous 
spectrum. 

Prior to [18], discretely decomposable restrictions tt\g> were known in 
some specific settings, e.g. the ^-correspondence for the Weil representation 
with respect to compact dual pair [TTJ, or when 7r is a holomorphic discrete 
series representation and G' is a Hermitian Lie group [14J. A systematic 
study in the general case including Zuckerman derived functor modules A q (X) 
was initiated by the author in a series of papers [20j |2TJ [22j [25j [27]. See 
P ESI 1201 EH ES] for a number of concrete examples of branching laws n\ G / 
in this framework, (33] for some application to modular symbols, [21] for the 
construction of new discrete series representations on non-symmetric spaces. 
See also the lecture notes [28] for a survey on representation theoretic aspects, 
and [251 12Z] for some applications. 

In this section, we give a brief overview of discretely decomposable re- 
strictions including some recent developments and open problems. 

5.1 Infinitesimally discretely decomposable restrictions 

Let us begin with an algebraic formulation. Suppose g' is a Lie algebra. 
Definition 5.1. A g'-module V is said to be discretely decomposable if there 
exists an increasing filtration {V n } such that V = IJ^Lo ^ n anc ^ eacn ^™ * s °f 
finite length as a g'-module. 
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In the setting where G' is a real reductive Lie group with maximal com- 
pact subgroup K', the terminology 'discretely decomposable' fits well if V is 
a unitarizable (g', K')-modvle, namely, if V is the underlying (g', i^')-module 
of a unitary representation of G'\ 

Remark 5.2 ([221 Lemma 1.3]). Suppose V is a unitarizable (g', i^')-module. 
Then V is discretely decomposable as a g'-module if and only if V is decom- 
posed into an algebraic direct sum of irreducible (g', .fT')- modules. 

We apply Definition 15. II to branching problems. Let G be a real reductive 
Lie group, and G' a reductive subgroup of G. We may and do assume that 
K is a maximal compact subgroup of G and K' := K D G" is that of G'. 

Definition 5.3. Let 7r be a unitary representation of G of finite length. 
We say the restriction tt\g> is infinitesimally discretely decomposable if the 
underlying (g, i^)-module n K is discretely decomposable as a g'-module. 

Here is a comparison between the category of unitary representations and 
that of (g, f^-modules: 

Conjecture 5.4. Let tt be an irreducible unitary representation ofG, and G' 
a reductive subgroup of G. Then the following two conditions on (G, G', tt) 
are equivalent: 

(i) The restriction tt\g> is infinitesimally discretely decomposable. 

(ii) The unitary representation tt decomposes discretely into a direct sum of 
irreducible unitary representations of G' . 

In general, the implication (i) =>- (ii) holds. Moreover, the branching law 
for the restriction of the unitary representation it to G' and that for the 
restriction of the (g, i^)-module tik to (g', K') are essentially the same under 
the assumption (i) (see [261 Theorem 2.7]). The converse statement (ii) =^ 
(i) remains open; affirmative results have been partially obtained by Duflo 
and Vargas [5] for discrete series representations tt, see also [26, Conjecture 
D] and [IS]. 

For the study of discretely decomposable restrictions, the concept of in- 
admissible restrictions is useful: 

Proposition 5.5. If the restriction 7t\k> is K' -admissible then both the con- 
ditions (i) and (ii) in Conjecture \5.4\ hold. 



Proof. See [221 Proposition 1.6] and [201 Theorem 1.2], respectively. □ 
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5.2 Analytic approach 



We now consider a criterion for the ^'-admissibility of a representation tt. 

Let K' be a closed subgroup of K. Associated to the Hamiltonian in- 
action on the cotangent bundle T*(K/K'), we consider the momentum map 



Then its image equals a/— 1 Ad*(K)(t') ± , where {t') ± is the kernel of the 

projection pr t ^ { / : I* — > (&')*, the dual to the inclusion I' C t of Lie algebras. 
The momentum set Ck{K') is defined as the intersection of Image fi with 
a dominant Weyl chamber C + (c y/— It*) with respect to a fixed positive 
system A + (t, t) and a Cartan subalgebra t of t: 



Here we regard t* as a subspace of t* via a i^-invariant non-degenerate bi- 
linear form on t. 

Next, let Ti be a i^-module. We write AS^(7r) for the asymptotic K- 
support introduced by Kashiwara and Vergne [T5], that is, the limit cone of 
the set of highest weights of i^-types in tt. AS^(7r) is a closed cone in C + . 

We are ready to state a criterion for admissible restrictions. 

Theorem 5.6. Let G D G' be a pair of reductive Lie groups, and take 
maximal compact subgroups K D K' , respectively. Suppose tt is an irreducible 
unitary representation of G. 

1) Then the following two conditions are equivalent: 



(ii) The restriction tt\k> is K' -admissible. 

2) If one of the equivalent conditions (i) or (ii) is fulfilled, then the re- 
striction tt\c is infinitesimally discretely decomposable (see Definition 
\5.3\) . and the restriction tt\g' is unitarily equivalent to the Hilbert direct 






(5.1) 



(i) C K (K>)nAS K {7r) = {0}. 



sum: 




with n n (r) < oo for any r G G' . 



t£G' 
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Outline of Proof. The proof of the implication (i) =>- (ii) was proved first by 
the author [2T], Theorem 2.8] by using the singularity spectrum of hyperfunc- 
tion characters in a more general setting where 7r is just a i^-module such 
that the multiplicity 

m^r) := dim Horner, it) 

is of infra-exponential growth. In the same spirit, Hansen, Hilgert, and Keliny 
[TU] gave an alternative proof by using the wave front set of distribution 
characters under the assumption that m 7r (r) is at most of polynomial growth. 
The last statement was proved in [211 Theorem 2.9] as a consequence of 
Proposition 15.51 See also [28]. □ 

The condition (i) in Theorem 15.61 is obviously fulfilled if Ck{K') = {0} 
or if ASk (7r) = {0}. We pin down the meanings of these extremal cases: 

1) Ck(K') = {0} -v^ K' = K. Then the conclusion in Theorem 15.61 2) is 
nothing but Harish-Chandra's admissibility theorem (see Fact 13. 4[ ). 

2) AS A '(tt) = {0} <=> dimvr < oo. 
5.3 Algebraic approach 

For a finitely generated g-module X, the associated variety V gc (X) is a sub- 
variety in the nilpotent cone Af gc of g^ (see [12]). In what follows, let X be 
the underlying (g, i^)-module of it G G and Y the underlying (g', i^')-module 
of r(Gff). 

We write pr g _j, g / : gj — > (q' c )* for the natural projection dual to q' c <->• gc- 

Theorem 5.7 (see [221 Theorem 3.1]). // Hom g ,(y,X) ^ {0} ; then 

p W (V gc (X))cV g ,(F). (5.2) 

Theorem 15.71 leads us to a useful criterion for discrete decomposability by 
means of associated varieties: 

Corollary 5.8. If the restriction X is infinitesimally discretely decomposable 
as a g' -module, then pr g ^ g , (V gc (X)) is contained in the nilpotent cone of g' c . 

Remark 5.9. An analogous statement to Theorem 15.71 fails if we replace 
Hony(r,X) ^ {0} by Hom G ,(r, tt| g ^ {0}. 
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Remark 5.10. Analogous results to Theorem 15.71 and Corollary 15.81 hold in 
the category O. See [52] . 

It is plausible that the following holds: 
Conjecture 5.11. The inclusion (15 .2p in Theorem 5.1 is equality. 

Here are some affirmative results to Conjecture 15.111 
Proposition 5.12. 

1) X is the Segal- Shale-Weil representation, and g' = g[ ©g' 2 is the com- 
pact dual pair in q = sp(n, M.). 

2) X is the underlying (g, K) -module of the minimal representation of 
0{p,q) (p + q even), and (g, g') is a symmetric pair. 

3) X is a (generalized) Verma module, and (g,g') is a symmetric pair. 

4) X = A q (X) and (g,g') is a symmetric pair. 

Proof. The first statement could be read off from the results in [38] by case- 
by-case argument though they were not formulated by means of Theorem 15.71 
See [34J for the proof of the second, and [32] for that of the third statement, 
respectively. The fourth statement is proved recently by Y. Oshima by using 
a T>- module argument. □ 



5.4 Restriction of ^4 q (A) to symmetric pair 

For the restriction of A^(X) to a reductive symmetric pair, our criterion is 
computable. Let us have a closer look. 

Suppose that (G, G') is a symmetric pair defined by an involutive auto- 
morphism a of G. As usual, the differential of a will be denoted by the same 
letter. By taking a conjugation by G if necessary, we may and do assume 
that a stabilizes K and that t and A + (t, t) are chosen so that 

1) tr°" := tfl t~ a is a maximal abelian subspace of t~ a , 

2 ) E + (^ r,T ) : = {Mi- ■ A G A + («,t)} \ {0} is a positive system of the 
restricted root system XX^* r<T )- 
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Then the momentum set Ck{K') coincides with the dominant Weyl chamber 
(C ^f^l(i- a )*) with respect to £ + (£,r ,T ). 



Let A(u n p) C v / -H* be the set of weights in u n p, and M + A(u n p) the 

closed cone spanned by A(ufl p). Then the asymptotic support AS^(^4 q (A)) 
is contained in M + A(u PI p). 

Theorem 5.13. The following six conditions on (g,0 <T ,q) are equivalent: 

(i) AJX) is non-zero and discretely decomposable as a g' -module for some 
A in the weakly fair range. 

(i)' v4 q (A) is discretely decomposable as a g' -module for any A in the weakly 
fair range. 



(iii) ^4 q (A) is non-zero and K' -admissible for some A in the weakly fair 



(hi)' Aj(A) is K' -admissible for any A in the weakly fair range. 

(iv) pr _ i , /(V 0c (A q (A))) is contained in the nilpotent cone of g' c . 

Proof. The equivalences (i) <=> (i)' and (iii) (iii)' are easy The implication 
(ii) =>- (iii) was first proved in [20J. Alternatively, we can use Theorem 15.61 
and the inclusive relation ASk(A|(^)) C 1R + A(uflp). This was the approach 
taken in [21] . Other implications are proved in [23J based on Theorem 15 .71 □ 

See [36] for the list of all such triples (g, g a , q). 
Remark 5.14. The implication (i) (iii)' and Theorem 15.61 show that 



if the restriction tt\g> is infinitesimally discretely decomposable for any ir^ ~ 





range. 



dimHom G /(r, 7t|g") < oo for any r G G' 



A,(X). 
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6 Appendix — basic properties of A q (A) 

This section gives a quick summary of basic properties on Zuckeman's derived 
functor modules and the "geometric quantization" of elliptic coadjoint orbits 
Ox in the following scheme: 

A G \J — lg* an elliptic and integral element 
C\+ Px — > 0\ a G-equivariant holomorphic line bundle 

f 

H^(0\, £\ +Px ) a Frechet representation of G 

7i\ a unitary representation of G 

There is no new result in this section, and the normalization of the pa- 
rameters and formulation follows the expository notes [231 EE] ■ See [TB] for 
a more complete treatment and references therein. 



6.1 Zuckerman derived functor modules 

Let G be a connected real reductive Lie group, g = t + p a Cartan decompo- 
sition of the Lie algebra of g, and 9 the corresponding Cartan involution. 

Let q be a ^-stable parabolic subalgebra of 0c- Then the normalizer 
L = N G (q) is a connected reductive subgroup of G, and the homogeneous 
space G/L carries a G- invariant complex structure such that the holomorphic 
tangent bundle T{G/L) is given as a homogeneous bundle Gx l(Qc/(\)- Let lc 
be the complexification of the Lie algebra I of L, and u the unipotent radical 
of q. Then we have a Levi decomposition q = [ c + u. We set p(u)(X) := 
| Trace(ad(X) : u -)■ u) for X 6 I 

We say a Lie algebra homomorphism A : [ — > C is integral if A lifts to a 
character of the connected group L, denoted by C\. Then C\ := G x Gx C\ 
is a G-equivariant holomorphic line bundle over G/L. For example, 2p(u) is 
integral, and the canonical bundle Q(G/L) := A top (T*(G/L)) is isomorphic 
to 

n(G/L) ~ C 2p(u) (6.1) 

as a G-equivariant holomorphic line bundle. The Zuckerman derived functor 
W i — y TZi^W ® ^-t>(u)) is a covariant functor from the category of ([,1(1 Jf)- 
modules to the category of (0, i^)-modules. We note that L is not necessarily 
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compact. In this generality, H. Wong proved in jH] that the Dolbeault 
cohomology groups 

H^G/L, C x ® n(G/L)) ~ #|(G/L, C x+2p(u) ) 

carry a Frechet topology on which G acts continuously and that 722(C\+p( u )) 
are isomorphic to their underlying (g, i-Q-modules. We set S := dimc(untc), 
and 

A q (A) :=^(C A+p(u) ). 

In our normalization, ^4 q (0) is an irreducible and unitarizable (g, -fT)-module 
with non-zero (g, i^)-cohomology [43], and in particular, has the same in- 
finitesimal character with that of the trivial one dimensional representation 
C of G. 

6.2 Geometric quantization of elliptic coadjoint orbit 

Let A G \/^lg*. We say that the coadjoint orbit 0\ := Ad*(G) • A is elliptic if 
A | j, = 0. We identify g with the dual space g* by a non-degenerate G-invariant 
bilinear form, and write X\ G y/— lg for the corresponding element to A. 
Then a.d(X\) is semisimple and all the eigenvalues are pure imaginary The 
sum of the eigenspaces for non- negative eigenvalues of — \/— 1 &d(X\) defines 
a ^-stable parabolic subalgebra q = tc + u, and consequently, the elliptic 
orbit 0\ carries a G-invariant complex structure such that the holomorphic 
tangent bundle is given by G x l (flc/q)- 

We set p\ := p(u). If A + p\ is integral, namely, if A + p\ lifts to a 
character of L, then we can define a G-equivariant holomorphic line bundle 
£ X + Px ■= G x L C x+Px over O x . 

Here is a brief summary of the important achievements on unitary repre- 
sentation theory in 1980s and 1990s on the geometric quantization of elliptic 
orbits due to Parthasarathy, Zuckerman, Vogan and Wallach (algebraic con- 
struction, unitarizability of Zuckerman derived functor modules A q (A)), and 
Schmid and Wong (realization in Dolbeault cohomology, in particular, the 
closed range property of the <9-operator) among others. See [161 12H] for the 
original references therein. 

Fact 6.1. Let A G \/—lg* be elliptic such that A + p x is integral. 
1) (vanishing theorem) H^(0\, £\+ Px ) = if j ^ S. 
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2) The Dolbeault cohomology group if| (0\, C\ +Px ) carries a Frechet topol- 
ogy, on which G acts continuously. It is the maximal globalization of 
1Zq(C\) = A q (\ — pa) in the sense of Schmid. 

3) (unitarizability) There is a dense subspace T-L in H^(0\, C\ +Px ) on 
which a G-invariant Hilbert structure exists. We denote by -K\ the re- 
sulting unitary representation on H. 

4) If A is in the good range in the sense of Vogan, then the unitary repre- 
sentation of G onH is irreducible and non-zero. 

Here, by 'good range', we mean that A satisfies 

(A + pi, a) > for any a e A(u, f) c ), (6.2) 

where t) is a fundamental Cartan subalgebra containing X\ and pi is half 
the sum of positive roots for A(lc, f)c)- (This condition is independent of the 
choice of f) and A + (l<c, t)c)-) 
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